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Abstract 



Problem-based learning (PBL) as a new instructional method is becoming increasingly popular. PBL is 
hypothesized to have a number of advantages on learning because it applies insights from cognitive 
learning theory and because it fosters a lifelong learning strategy. As in all learning programs there are 
individual differences between students. The present study investigates individual differences in student’s 
problem-based learning behavior (PBL behavior) and its effect on achievement. 

PBL behavior is theoretically conceptualized as an integrated concept of two dimensions of students’ 
behavior: The study approach that students use, the ‘style’ of their PBL behavior, and the kind of 
‘activity’ they show when they use the PBL systematic working procedure. In this, the student’s behavior 
both during tutorial group meeting as well as during individual study is regarded as part of PBL behavior. 
The results indicate that achievement is significantly influenced by the activity of PBL behavior after 
correction for more general test preparation behaviors. The style of PBL behavior did not yield a 
significant contribution. Given the statistically sigmficant relationship found here with one of the PBL 
behavior dimensions, it would be valuable to investigate further the effect of both dimensions PBL style 
and activity on achievement, both in isolation and in combination. 

Introduction 

According to Barrows & Tamblyn (1980) problem-based learning (PBL) may improve students’ general 
problem-solving skills, enhance integration of basic concepts into specific problems, foster the 
development of self-directed learning skills, and strengthen intrinsic motivation. In PBL one of the major 
objectives is self-directed learning (Williams, 1992). Williams argues that in die information age, schools 
have the responsibility to ensure that their graduates have skills for lifelong learning: how and where to 
obtain information, how to evaluate it, and how to apply knowledge to solve problems. 

PBL, referring to the educational system or philosophy, is hypothesized to have a number of advantages 
over traditional approaches to teaching and learning. In small group discussion, students solve a problem 
within a context. The systematic working procedure students use requires cogmtive processes which 



stimulate learning (Schmidt, 1983). While defining and analyzing problems students activate their prior 
knowledge. This focuses the learning effort and facilitates the understanding of new concepts to be 
mastered. By analyzing, organizing potential explanations and informing one another before and after 
individual study, students elaborate, structure and restructure their prior knowledge and synthesize new 
knowledge. Group discussion, explaining the phenomena to other students and listening to the 
explanations from other students, will lead to an enrichment of the cognitive structures of the participants. 
The knowledge already available and the new knowledge becomes tuned to the specific context of the 
problem. In addition, through the group discussion, through the professional authenticity of the problem 
and through the formation of individual learning objectives epistemic curiosity is aroused. In a recent 
review Norman and Schmidt (1992) presented empirical evidence for these claims. 

Despite these presumed advantages of PBL, little is known about individual differences in student s PBL 
behavior and its effect on knowledge building and achievement. This stimulated the present study in which 
the quality of learning behavior in PBL is related to outcome, i.e. student achievement. 

One can define PBL behavior as the extent to which students study and behave according the principles of 
PBL. In PBL, the separate elements of the systematic working procedure are complementary to each other. 
Both student behavior during group discussion proceding and following individual study and the 
individual study in itself are instrumental in learning and knowledge building (De Grave, et al., 1996; 
Schmidt, 1993). 

In several studies investigating learning behavior, a deep or meaning oriented study approach turned out to 
be more effective than a surface approach (e.g. Trigwell & Prosser, 1991; Van Rossum & Schenk, 1984). 
In a study specifically conducted in a PBL context Van Berkel, Nuy and Geerligs (1995) found similar 
tendencies. Achievement tended to improve when learning processes were used which could be 
characterized as meaning oriented, whereas poorer achievement results tended to be found when 
reproduction oriented learning processes were applied. Entwistle (1988) concludes that when students use 
a deep level approach to reach understanding, their behavior is characterized by logically examining 
evidence step by step. These students are cautious in accepting generalizations and have a broad focus on 
the outline of ideas and their interconnections. Students using a surface approach easily accept 
generalizations. They focus on tasks and pieces of information in isolation without seeking for successful 
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outcomes. The educational context of PBL trains students to use a systematic procedure resembling a deep 
level approach. Coles (1985) and Newbie and Clarke (1986) showed that students of a problem-based 
school scored higher on a deep approach scale and lower on a surface approach scale than students of 
traditional schools. 

The behavioral characteristics of deep and surface approaches which are used during individual study 
(Entwistle, 1988) may also be used during group interaction in the tutorial groups. Examples of deep 
processing behavior during problem analyses in the tutorial group include carefully attending to 
contradictory information of other students, attempting to understand an alternative explanation advanced 
by other students, elaborating on relations between information of other students and considering the 
fullest range of alternative explanations. In addition to differences in learning behaviors applied, 
individual differences in group interactions are manifest. Some students are much more active than others 
in their group interaction. They will try to solve the problem, ask for specific information, summarize 
content, discuss with others, clarify difficult material to others, and check predictions. 

We therefore distinguish theoretically two dimensions of PBL behavior by students. The first is a deep or 
surface study approach that students use, which will be called ‘style of PBL behavior here. The second 
one is the kind of activity of PBL group behavior, consisting of active or passive behavior in interacting in 
group discussion. By combining the poles of these two dimensions four combinations of PBL behavior are 
possible; surface active, deep active, surface passive or deep passive PBL behavior. The latter PBL type 
seems a contradiction in terms. However, De Grave, Boshuizen and Schmidt (1996) showed that even 
when students do not interact (passive PBL behavior), they can utilize deep learning processes. 

In the present study, PBL will be regarded as an integrated concept of students’ behavior both during 
tutorial group meetings as well as during individual study. The impact of PBL behavior, both style and 
activity, will be related to achievement. Since we know that within any educational and instructional 
design achievement tests are a strong stimulus for all kinds of learning behavior (e.g. Newbie & Entwistle, 
1986; Frederiksen, 1984), we have to control for ‘common’ learning behavior which is less intrinsically 
related to PBL, such as test-directed studying. Therefore the central purpose of the present study is to 
investigate the unique impact of PBL behavior, defined as appropriate individual and tutorial group 
behavior, on student achievement after correcting for other non-PBL behavior. 



Method 



Subjects. The study was conducted at the Faculty of Health Sciences of the University of Maastricht, 
the Netherlands. In total 164 of 174 randomly selected second, third and fourth year students participated 
(response rate 94.3%). 

Materials. The materials used in this study consisted of a questionnaire about problem-based learning 
behavior, a small questionnaire about non-PBL behavior and an achievement test. 

Problem-based learning behavior was measured using 24 ‘vignettes’ consisting of situational 
descriptions of problem-based learning behavior. Each vignette represented one of the four bipolar 
combinations of ‘style’ and ‘activity’ of problem-based learning behavior (deep active, surface active, 
deep passive or surface passive PBL behavior). Each vignette contained a set of behaviors with activities 
or interactions which characterize one of the PBL types within one of the stages of the learning process as 
is systematically applied in Maastricht (the seven-jump; clarify terms, define the problem, analyze the 
problem and organize potential explanations, formulate learning objectives, individual study, report the 
results in the tutorial group by synthesizing the newly acquired information). So, for each stage four 
situational descriptions were constructed, representing one of the four types of PBL behaviors. The 
vignettes were constructed on the basis of exploration of the literature (De Grave, 1993; Moust, Bouhuijs 
& Schmidt, 1989) and the advice of a review panel of educationalists with expertise in PBL. Figure 1 
shows an example of the situational descriptions for ‘formulating student generated learning issues’. 
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PBL behavior 


style of PBL behavior 


surface 


deep 


activity 
of PBL 
behavior 


passive 


“Particularly, I listen to the ideas of my 
colleague students. I, myself, mostly wait before 
giving ideas. I think about ideas, but for me it’s 
difficult to determine what the most important 
topic in a discussion, and what the most 
important learning objectives are. For me, it is 
also difficult to judge the proposals of others on 
clarity, specificity and relevance”. 


“Particularly, I listen to the suggestions for 
learning objectives of my colleague students. I see 
which issues require further exploration. I 
generally wait for the suggestions of others. 
Eventually, the final learning issues are very 
similar to the ones I thought were relevant”. 


active 


“Usually I state my ideas for learning goals. 
However, I find it difficult to define topics after 
the brainstorm and analysis in order to formulate 
learning objectives. I also find it difficult to 
formulate them in a clear and workable manner. 
My contribution often facilitates other students 
to make a more explicit and clear proposal”. 


“Normally, after brainstorm and analyses I’m able 
to formulate the topics of the learning objectives. 
I’m also able to formulate them in a clear and 
workable manner. I generally report the ideas I 
have. The learning goals finally formulated by the 
tutorial group cover the ones I thought were of 
interest”. 


Figure 1 P 


3L- vignettes for “formulating student generated learning issues” 



Students marked on a 5-point Likert scale (1= (hardly) not, or 5=(almost) totally) the extent to which they 
recognized themselves in the described situation. The average score across the six vignettes for each of the 
PBL-types was calculated. Students were classified as belonging to the PBL-type on which they had the 
highest average score. In order to arrive at an estimate of proficiency on each of the dimensions, activity 
and style, interval values were calculated for each student for each dimension' . 

Non-problem-based learning behavior was assessed by three questions which were considered relevant 
to achievement testing. One question asked about the “time spent on individual study for the achievement 
test” not related individual study for the tutorial group, and a second about the “total time spent on 
individual study” during a block, responses recorded in hours. A third question asked about students’ 
estimation of the extent to which they “explicidy studied for the achievement test , measured on a 5 -point 



* A dimension score was defined by the difference of the sums for each pole of that dimension across the poles of 
the other dimension. For example; To calculate ‘style’, average scores for the two PBL-types containing surface, 
surface passive and surface active, are added. The same is done for the two PBL-types containing deep, deep passive 
and deep active. Style is calculated by the difference between the values for surface and deep. 
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Likert scale using single vignettes giving a situational description. 

Achievement was measured by the progress test, an end-of-course test, which is congruent with the 
educational principles of problem-based learning. In the Health Sciences Faculty, this comprehensive test 
consists of 400 items in the true-false format with a ‘don’t know’ option. To correct for guessing formula 
scoring is used; the overall score is the number of correct answers minus the number of incorrect answers, 
the true/false score. The ‘don’t know’ option is scored neutral. Three times a year, a newly constructed 
form of the test is administered to all students in the curriculum, regardless of their year. This test 
measures functional knowledge and has been proven not to discourage self-directed learning and not to 
reward ‘study for the test’ behavior (Blake, Keane, Norman & Mueller, 1994; Van der Vleuten, Verwijnen 
& Wijnen, 1996). 

Procedure. At the beginning of the academic year 1995/1996 students were asked to fill out the 
questionnaire about their learning behavior over the previous academic year. Students’ answers were 
related to the test score on the progress test preceding the questionnaire. For 22 students scores on the 
progress test were missing. 

Analyses. Descriptive statistics will be given for style and activity and each combination of PBL style 
and activity behavior. Using regression analyses the unique impact of the style and activity of PBL 
behavior on students’ achievement was examined after having entered the moderating variables year of 
training and non-PBL behavior. Year of training is entered in the equation as three dummy variables with 
value ‘ 1 ’ for participant of specified year and value ‘0’ for non-participant of specified year. 

Results 

Table 1 presents means and standard deviations of the calculated interval values on the dimensions 
activity and style for each of the combination of PBL style and activity behaviors, the PBL-types. 
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Table 1 Means and standard deviation on the interval value for the dimensions style and activity for each 



of the PBL-types 



PBL-type 


PBL-activity 


PBL-style 




mean 


st.dev 


mean 


St.dev 


N 


deep active 


1.89 


0.99 


2.49 


1.04 


74 


surface active 


-0.25 


0.62 


1.63 


1.13 


33 


deep passive 


0,69 


0.88 


-0.59 


0.97 


34 


surface passive 


-1.13 


0.50 


-1.40 


0.75 


12 



Deep active students have the highest interval values on both the PBL dimensions style and activity 
against lowest interval values by surface passive students. Strangely enough, do deep passive students 
have higher interval values for the activity dimension than surface active students. Also, surface active 
students do have higher interval values for style than deep passive students. 

Table 2 reports means and standard deviations of the scores on the progress test for each of the PBL- 
types, the combination of style and activity behaviors 



Table 2 Means and standard deviation on the progress test for each combination of PBL style and 



activity behaviors 







2 




year of training 
3 




4 






total 




PBL-type 


mean 


st.dev 


n 


mean 


st.dev 


n 


mean 


st.dev 


n 


mean 


st.dev 


n 


deep active 


57.3 


12.2 


20 


61.8 


18.3 


13 


67.2 


20.8 


26 


62.6 


18.0 


59 


surface active 


39.9 


9.9 


9 


58.9 


19.0 


7 


65.4 


11.9 


16 


56.8 


16.9 


32 


deep passive 


54.8 


16.1 


12 


57.0 


27.1 


8 


68.9 


16.0 


11 


60.4 


19.8 


31 


surface passive 


41.0 


10.5 


5 


55.0 


5.7 


2 


64.5 


15.0 


4 


52.1 


15.5 


11 



Highest achievement scores are obtained by students with deep active behavior and lowest scores for 
students with surface passive behavior. In the highest year of training this effect is less pronounced. 

In table 3 the correlation matrix for the non-PBL behavior variables, the PBL behavior variables and the 
achievement score is presented. The dummy variables for year of training are not reported. 
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Table 3 Correlation matrix for aehievement seore, non-PBL behavior and PBL behavior 





achievement 


hours study 
for block 


hours study 
for test 


explicit study PBL-style 
for test 


hours study for block 


0.12 








hours study for test 


-0.19* 


0.09 






explicit study for test ‘ 


0.27*** 


0.04 


-0.57*** 




PBL-style 


0.11 


0.18* 


■0.05 


0.15* 


PBL-activity 


0.23** 


0.03 


-0.11 


0.18* 0.51*** 



‘ a lower score means more explicit individual study behavior due to achievement testing, which is less desired within PBL 
* p s 0.05; •• p s 0.01; *•* ps 0.001 

The correlations of PBL behavior and non-PBL behavior with achievement range from -0. 19 to 0.27. The 
activity dimension of PBL behavior shows a correlation of r=0.23 (p^ .0 1) with achievement, whereas the 
style dimension of PBL behavior does not (r=0. 1 1, p>.05). When it comes to the non-PBL variables, 
particularly the correlation of the variable ‘explicit study for the achievement test’ with achievement is 
considerable (r=0.27, p^ .001). It should be noted that this correlation means that students who do not 
explicitly study for the test score higher than students who explicitly do study for the test. The variables 
representing activity and style of PBL behavior are highly correlated (r=0.5 1, p^ .00 1). 
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Table 4 Regression model with the dependent variable in the equation achievement score, the 

independent moderating variables year of training and non-PBL behavior, and the independent 
variables PBL behavior 





B 


se B 


p 


T 


P 


moderating variables in the equation (method enter) 












year 1 (dummy) 


-7.7 


1.5 


-0.8 


-5.1 


0.00 


year 2 (dummy) 


-6.2 


1.5 


-0.5 


-4.2 


0.00 


year 3 (dummy) 


-4.3 


1.4 


-0.4 


-3.1 


0.00 


hours of individual study during block 


-0.3 


0.0 


-0.1 


-0.8 


0.45 


hours of individual study for the test 


0.0 


0.2 


0.0 


0.2 


0.83 


explicit individual study behavior for achievement testing 


0.5 


0.4 


0.1 


1.5 


0.14 


impact of PBL behavior (method stepwise) 












in the equation; activity* 


0.6 


0.3 


0.2 


2.2 


0.03 


Constant 


19.6 


2.6 




7.6 





' the variable PBL-style reveals no significant impact in explaining the regression model, with T— 0.83 and p-0.41 



The regression analyses revealed (table 4) that almost 25% of variation in test scores is explained by year 
of training and non-PBL behavior (R^=.254, adjusted R^=.22 1). Not surprisingly, most of it can be 
attributed to year of training. Stepwise addition of the two PBL behavior variables, style and activity 
showed that only the activity dimension significantly contributes for achievement. The explained variance 
increases to about 28% (R^=.279, adjusted R^=.242). 



Discussion and conclusion 



This study investigated the impact of PBL study and group behavior on student achievement. 

Achievement is significantly influenced by the activity of PBL behavior: Being more active and taking the 
initiative in using a systematic PBL procedure in the tutorial group improves achievement. However, the 
incremental amount of explained variance is limited (an increase of 3% explained variance). This might be 
a reflection of a relatively homogeneous PBL population. Students in a problem-based school already 
show more deep learning behavior (Coles, 1985; Newbie & Clarke, 1986) and are more actively involved 
in the learning process than students in traditional schools. Furthermore, other studies have shown that 



even with many study behavior variables only a limited amount of achievement variance can be explained 
(Meerum Terwogt-Kouwenhoven, 1990; Van Overwalle, 1985). 

The other PBL behavior dimension, style of PBL behavior, did not significantly contribute to 
achievement. This dimension is fairly highly intercorrelated with activity of PBL behavior and comparison 
of the raw scores for style and activity of PBL behavior across the PBL types showed that when students 
score high on one of the dimensions, they also score high on the other dimension. 

The significance of the activity dimension does not confirm the findings of a study by Moust et al. 

( 1 986), which failed to demonstrate that the quantity of students’ contribution during activation of prior 
knowledge, expressed as the number of verbalizations, was related to achievement. Assuming similarities 
between the number of verbalizations with some characteristics of active PBL behavior, the contradiction 
is even stronger. Because the authors also expected positive effects of activity, they proposed that the 
more silent students were involved in what they called ‘covert elaboration’, using deep learning 
approaches without verbalizing them. 

In previous studies (e g. Trigwell & Prosser, 1991; Van Rossum & Schenk, 1984) results showed 
favorable achievement scores for students with learning behavior which can be characterized as deep 
oriented. Also the study of Van Berkel et al. (1995), carried out in a PBL-context, reported similar 
findings, although they were unable to find statistically significant relationships. In this perspective, it 
seems of additional theoretical value to define PBL behavior as a complex concept which includes both 
behavior in the tutorial group and during individual study, and to distinguish the characteristics of style 
and activity of that behavior. 

An interesting result was the significant inverse relationship between explicit test-directed studying and 
the achievement score on the progress test. Appaiently, studying for the progress test is no guarantee for 
better achievement. Even the opposite seems the case: students not preparing for the test perform better. 
This is inline witlithe intention of this testing procedure, described by Van der Vleuten, Verwijnen and 
Wijnen (1996): Progress testing does not reward ‘study for the test’ behavior and does not discourage 
self-directed learning. Furthermore, the pattern of behavior of not preparing for the test positively 
correlates with deeper and more active PBL behavior. 

It should be noted that the questionnaire in the present study was an initial attempt to measure PBL 
O 10 
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behavior both during individual study and during tutorial group interactions along the dimensions style 
and activity. Instead of asking for global self-ratings, which usually are poor predictors of actual behavior 
(Gordon, 1991), this instrument confronts students with concrete and authentic PBL-situations and 
requires them to judge recognition of their own behavior in these situations. The validity of this approach 
needs further refinement and research. Given the statistically significant relationship found here with one 
of the PBL behavior dimensions, it would be valueable to investigate more thoroughly the effect of the 
dimensions PBL style and activity on achievement, both in isolation and in combination. 
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